Moving frame display

ABSTRACT

An apparatus and method of providing a display that simplifies presentation of related video frames to enable a user to better see and interpret videos exhibiting a high degree of localized apparent image jitter. The present invention includes embodiments directed towards presentation of image sequences in a contextual display responsive to a contextual map for controlling apparent jitter and towards compositing for production of contextual maps by co-locating content alignment attributes which results in non-alignment of individual frames.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/350,805 filed 15 Jan. 2012, the contents of which is hereby expresslyincorporated by reference for all purposes.

BACKGROUND OF THE INVENTION

The present invention relates generally to presentation of a series ofrelated images, and more particularly, but not exclusively, topresentation of those images in a larger consistent frame of referencethat dampens localized apparent image jitter.

Many types of portable electronic devices allow a user to capture aseries of images, the uses of these devices are causing significantgrowth in a quantity of digital photographs that are casually acquiredand displayed. In most cases these images are captured under non-idealconditions, with non-ideal acquisition equipment, by non-professionalimagers. For example, in situations such as filming with a hand-heldcamera: a) from a moving vehicle, b) during sporting activities, c) in ahigh-vibration environment, d) by a user with novice filming skills,and/or e) while imaging in extreme high-stress situations (e.g., whilerunning on foot), many image sequences exhibit, when played back usingconventional players, a high degree of unwanted motion or jitter causedfrom randomized motion of the camera about the desired field-of-view.Even videos acquired in normal conditions show a certain amount ofunwanted shaking, and under some conditions professional imagers willproduce image sequences with undesirable amounts of shaking.

There exist several methods for stabilizing a video image, either as theimage is being acquired, or as a post-processing step before display.Some cameras provide mechanical image stabilization at the time theimage is acquired, utilizing inertial measuring devices such asaccelerometers or gyroscopes to provide information to calculate cameramotion in real time. Digital techniques can be employed that typicallyinvolve calculating image motion based on matching regions within theimages themselves that are assumed to represent stable backgroundfeatures. These techniques suffer from limitations of accuracy,reliability, applicability, and computation time (not to mention energy,size, and cost considerations) and tend to work most effectively onlywhen the jitter motion is small. These stabilization techniquesgenerally involve some loss or compromise of the photographicinformation that is available in the raw, unprocessed image data. Thesesolutions and techniques treat the randomized noise effects of thecamera motion as undesirable distortion and interference to beeliminated. They are practically useless in cases where the cameramotion is so great that, for example, one image frame may have movedcompletely outside the bounds of the previous image frame. The resultingpresentation is often extremely awkward and difficult to see, anddemands significant extra effort to interpret.

There exists a need for an apparatus and method of providing a displaythat simplifies presentation of related video frames to enable a user tobetter see and interpret videos exhibiting a high degree of localizedapparent image jitter.

BRIEF SUMMARY OF THE INVENTION

Disclosed is an apparatus and method of providing a display thatsimplifies presentation of related video frames to enable a user tobetter see and interpret videos exhibiting a high degree of localizedapparent image jitter. The present invention includes embodimentsdirected towards presentation of image sequences in a contextual displayresponsive to a contextual map for controlling apparent jitter andtowards compositing for production of the contextual maps. Embodimentsof the present invention are useful for videos having small amounts ofsuch jitter as well.

A presentation system includes a parser producing a set of discreteframes from a series of related image frames representing a scene; acompositor, responsive to the set of discrete frames, generating acontext map that identifies a plurality of transformational attributesbetween each discrete frame of the set of discrete frames; and apresenter, responsive to the context map, generating a presentation ofthe set of discrete frames within a display field larger than a framesize of each of the discrete frames wherein the presentation includesoverlapping non-aligned discrete frames collectively reproducing thescene.

A computer-implemented presentation method using a microprocessorincludes a) producing, using the microprocessor, a set of discreteframes from a series of related image frames representing a scene; b)generating, responsive to the set of discrete frames and using themicroprocessor, a context map that identifies a plurality oftransformational attributes between each discrete frame of the set ofdiscrete frames; and c) generating, responsive to the context map, apresentation of the set of discrete frames within a display field largerthan a frame size of each of the discrete frames wherein thepresentation includes overlapping non-aligned discrete framescollectively reproducing the scene.

Humans and many other animals do not look at a scene with a fixedsteadiness; instead, a focus of the eyes changes quickly, locatinginteresting parts of a scene and building up a mental, three-dimensional‘map’ corresponding to the scene. These rapid eye movements are calledsaccades. One reason for the saccadic movement of the human eye is thatthe central part of the retina—known as the fovea—plays a critical rolein resolving object details. A human's saccades are very fast. Spatiallocation of visual information is preserved with great fidelity amidthese rapid movements of the eyes.

The present invention includes apparatus, method, and computer-programproduct to present a series of images exhibiting some degree of apparentcamera motion when viewed in a conventional manner, in a manner thatnaturally engages the inherent capacities of the eye to capture thingsin motion in relation to their environment, and creates an effectivedisplay of the motion picture without a destabilizing effect oflocalized apparent image jitter that is the usual result of the cameramotion.

A video playback method for rendering a series of frames, a first frameof the series of frames sharing a content alignment attribute with asecond frame of the series of frames, wherein the content alignmentattribute has a different apparent location relative to a visiblecontent of the first frame than it does relative to a visible content ofthe second frame, the method including (a) rendering the first frame ofthe series of frames, the rendering step (a) placing the first framewithin a presentation system which positions the content alignmentattribute of the first frame at a particular location; and thereafter(b) rendering the second frame of the series of frames, the renderingstep (b) placing the second frame within the presentation system toposition the content alignment attribute of the second frame with apredefined relationship to the particular location wherein the secondframe is non-aligned with the first frame.

A video playback system rendering a series of frames, the series offrames sharing a plurality of content alignment attributes wherein asubset of content alignment attributes associated with a particularframe of the series of frames each have a different apparent locationwith respect to a visible content of the particular frame as compared tocorresponding shared content alignment attributes of the subset ofcontent alignment attributes with respect to a visible contentassociated with another frame of the series of frames includes apresenter sequentially rendering the series of frames on a presentationsystem, the presenter placing each frame as it is sequentially renderedin the presentation system wherein its content alignment attributes havea predefined relationship with respect to the content alignmentattributes of previously rendered frames wherein the visible content ofsubsequently rendered frames is non-aligned with the visible content ofpreviously rendered frames.

A compositing system for a series of frames, each frame of the series offrames sharing a content alignment feature wherein the content alignmentfeature in the series of frames has a different apparent location withrespect to a visible content of adjacent frames in the series of framesincludes a compositor processing the series of frames and generating acontext map therefrom, the context map identifying a plurality ofrendering parameters for each frame of the series of frames that would,when the series of frames are rendered responsive to the plurality ofrendering parameters, produce a consistent placement of the contentalignment feature within a virtual reference space as each frame wouldbe rendered while non-aligning individual frames of the series of framesas they are rendered.

A presentation system including a parser producing a set of discreteframes from a series of related image frames representing a scene; acompositor, responsive to the set of discrete frames, generating acontext map that identifies a plurality of transformational attributesbetween each discrete frame of the set of discrete frames; and apresenter, responsive to the context map, generating a presentation ofthe set of discrete frames within a display field larger than a framesize of each the discrete frame wherein the presentation includesoverlapping non-aligned discrete frames collectively reproducing thescene.

A computer-implemented presentation method using a microprocessor, themethod including a) producing, using the microprocessor, a set ofdiscrete frames from a series of related image frames representing ascene; b) generating, responsive to the set of discrete frames and usingthe microprocessor, a context map that identifies a plurality oftransformational attributes between each discrete frame of the set ofdiscrete frames; and c) generating, responsive to the context map, apresentation of the set of discrete frames within a display field largerthan a frame size of each the discrete frame wherein the presentationincludes overlapping discrete frames collectively reproducing the scene.

The image composite, which selectively includes the camera motion, asdisplayed by embodiments of the present invention, is received by theeye in a manner that more closely approximates how the eye wouldactually and naturally receive the picture in the real world as if fromthe point of view of the image capture device. This is partially due tonot requiring the eye to spend the effort and time to interpret eachframe and infer the alignment of each frame of a series of frames asconventional systems do. The alignment of the frames, which means thatreference points shared (expressly or inherently) between the frames aresubjected to different placement, which is perceived as jitter.Embodiments of the present invention use the reference points asanchoring attributes, which results in non-alignment of the framesthemselves, thus the presentation system has a rendering area largerthan any single frame.

Other features, benefits, and advantages of the present invention willbe apparent upon a review of the present disclosure, including thespecification, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying figures, in which like reference numerals refer toidentical or functionally-similar elements throughout the separate viewsand which are incorporated in and form a part of the specification,further illustrate the present invention and, together with the detaileddescription of the invention, serve to explain the principles of thepresent invention.

FIG. 1 illustrates a view of a representative actual scene from which aseries of frames are recorded;

FIG. 2(a)-FIG. 2(g) illustrate individual frames of a series of framescaptured from the representative actual scene of FIG. 1;

FIG. 3(a)-FIG. 3(g) illustrate the series of frames shown in FIG.2(a)-FIG. 2(g) rendered on a player in conventional fashion;

FIG. 4 illustrates a parsing of the representative actual scene toproduce the individual frames of the series of frames illustrated inFIG. 2(a)-FIG. 2(g) and FIG. 3(a)-FIG. 3(g);

FIG. 5(a)-FIG. 5(g) illustrate the series of frames shown in FIG.2(a)-FIG. 2(g) rendered on an image presentation device;

FIG. 6 illustrates a parsing of the representative actual scene toproduce the individual frames of the series of frames illustrated inFIG. 2(a)-FIG. 2(g) and FIG. 5(a)-FIG. 5(g);

FIG. 7(a)-FIG. 7(k) illustrate the series of frames shown in FIG.2(a)-FIG. 2(g) rendered on an image presentation device in which eachindividual frame has a limited dwell time;

FIG. 8(a) and FIG. 8(b) illustrate rendering of individual frames fromthe series of frames shown in FIG. 2(a)-FIG. 2(d), with a 3 frame dwell,and including an outline border of currently rendered frames;

FIG. 9(a) and FIG. 9(b) illustrate rendering of individual frames fromthe series of frames shown in FIG. 2(a)-FIG. 2(d), with a 3 frame dwell,and including a current frame indicator for the most recently renderedframe;

FIG. 10 illustrates a schematic block diagram of a display system; and

FIG. 11 illustrates a flowchart for a display process.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention provide a display that simplifiespresentation of related video frames to enable a user to better see andinterpret videos exhibiting a high degree of localized apparent imagejitter. The following description is presented to enable one of ordinaryskill in the art to make and use the invention and is provided in thecontext of a patent application and its requirements.

Various modifications to the preferred embodiment and the genericprinciples and features described herein will be readily apparent tothose skilled in the art. Thus, the present invention is not intended tobe limited to the embodiment shown but is to be accorded the widestscope consistent with the principles and features described herein.

For purposes of the present disclosure, the following terms have thefollowing meanings. An actual scene refers to the real-world scene fromwhich frames are recorded. A composite scene is a rendering of theframes in a boundless (or effectively boundless) space to produce arepresentation of the actual scene. A frame is one of a series ofindividual images comprising a sequence of images (including “video” andthe like) that share a common temporal or geographic relationship, andwhich include one or more common anchors or reference points, some ofwhich may be virtual and exist outside a visible content of the frame.Each frame has a perimeter, or boundary, that surrounds all or a portionof the visible content. Apparent jitter addressed by preferredembodiments of the present invention result from alignment of the frameperimeters as each frame is rendered, with the common anchor orreference points having a different relative location within the frames.A presenter renders some or all of the composite scene and preserves therelationship between selected ones of these anchors as each frame issequentially rendered, such as by co-locating the anchoring content,which process non-aligns the frame boundaries. Video offers an inherenttemporal orientation of each frame in relationship to an immediatelypreceding frame and an immediately following frame, and successiveframes often share one or more anchor attributes (though it is notalways the case that the anchors are within the visible area of one ormore of the frames). Video recording of an actual scene also offers aninherent geographic orientation of the frames. Jitter, as that term isused herein, includes the effect of the discontinuous or inconsistentplacement of these anchors when these frames are rendered with alignedboundaries as when displayed in a conventional display.

A display field of the presenter refers to a virtual plane upon whichframes are rendered in a position, size, and orientation correspondingto the position, condition, and orientation of the imager at the actualscene when each frame was recorded. In some embodiments, one way ofunderstanding the effect is that an initial frame is placed in thecomposite scene, which fixes many of the anchors and reference points,including shared anchors and reference points in frames yet to berendered. Frames are rendered in sequence, one after another, and theposition, size, orientation, and other attributes of aframe-to-be-rendered are determined by the anchors and references itincludes that are shared by the frames that have been previouslyrendered. The frame, when rendered, matches the anchors and references,which can result in significant displacement and transformation of eachframe relative to previously rendered frames. This is one reason thatthe composite scene is virtual and effectively boundless as the framesare appropriately positioned in an internally consistent reference framehaving consistent and continuous anchors and references.

A display rectangle refers to a visible region of a display such as acomputer screen, usually rectangular and fixed, in which a specificportion of the display field is displayed, and a border, boundary,reticule, or other identifying feature may be added to highlight themost-recently rendered frame. In the preferred embodiment, anchors andreferences are identified during compositing and a contextual maprecords how a frame is rendered with respect to previously renderedframes. It is also the case that anchors and references may exhibitmotion or other change over time. Anchoring attributes are notnecessarily static and may move, and for practical reasons, it may benecessary or desirable to define specific anchors among a series ofactual frames as, in one interpretation, each actual frame may haveinfinitely many anchors shared by all subsequently captured images(though practically some sets of frames are viewed as independent fromother sets).

FIG. 1 illustrates a view of an actual scene 100 from which a series offrames are recorded. To simplify the discussion and understanding ofpreferred embodiments of the present invention, actual scene 100 isshown as a panorama view of a landscape having static anchors. Thepresent invention is not limited to such environments as it isapplicable to fully dynamic environments as well as hybrid environmentsincluding both static and dynamic elements. In some cases, an anchorreference or references for the actual scene will be static (such aswith the landscape depicted in actual scene 100) or may be dynamic, suchas use of a motive element (e.g., an overhead plane, a boat, an animal,a bicyclist, and the like) about which the user centers the frames whilerecording the motive element panning across other static or dynamicelements). It is not always the case that an anchor reference existswithin the visible portion of any frame, embodiments of the presentinvention may, in appropriate cases, use non-visible anchoringattributes (e.g., positional information like longitude/latitude,gyroscopic, accelerometer, and other absolute or relative coordinatedevelopment models).

FIG. 2(a)-FIG. 2(g) illustrate individual frames 200(a)-200(g) of aseries of frames captured from actual scene 100 shown in FIG. 1. Eachframe 200 _(x) is a portion of actual scene 100 recorded using animager, such as a camera, video recorder, or the like, to record apanoramic scan of the landscape. Each frame 200 _(x) includes a fixedrectangular profile determined by the imager having a content that isaffected by optical and mechanical characteristics of the imager as wellas mechanical conditions of the operator as the frames are recorded. Forexample, many imagers have scaling (e.g., zoom) options and the relativerelationship between actual scene 100 and an instantaneous field-of-viewof the imager when frame 200 _(x) is recorded will result in non-uniformrelative transitions when reconstructing the series of frames accordingto the temporal/geographical context. For each frame 200 _(x), theimager may be tilted or rotated differently, held at a different height,or taken from a different vantage point. For example, the operator maybe running and changes in the absolute and relative field-of-view due tomotion of the videographer and of the imager are superimposed over thedesired target of the series of frames, which itself may be rapidlychanging. The collective aggregation of motion superimpositions canproduce a chaotic series of images when played back using a conventionalsystem because the fixed rectangular profiles are what is typicallyaligned during rendering in a conventional system which causes thediscontinuous motions (e.g., jitter) of identifiable features (which canserve as anchor). Embodiments of the present invention help to smoothout all these motions by rendering frames with anchors having apredefined relationship to each other (typically co-located) andallowing the fixed rectangular profile to be non-aligned with otherframes.

Each imaged instantaneous field-of-view is captured in the rectangularframe which captures both the intended frame element as well as thesuperimposed imager effect. When played back using a conventionalplayer, the series of frames are presented in a rectangular displayconfigured to show each frame as it was captured, but normalized to theorientation of the player. This is what imparts the apparent jitter tothe image as the anchor locations between successive rendered frames isnot preserved. Actual scene 100 is not (under normal circumstances)shaking but the artifice of the capturing and rendering in conventionalmanner superimposes the motion of the imager onto frames 200 _(x) sothat, in the conventional player, there is apparent jitter whenrendering frames 200 _(x). As discussed above, the conventional solutionis to attempt to compensate and neutralize the jitter which solution isdesigned to smooth out the field-of-view distortions, but which alsoeliminates some of the information contained within the sequence, forexample, the motion of the imager. In some conventional systems,stabilizers (and stabilizer processes applied to a series of capturedimages) intentionally put to waste portions of the original full fieldof view by cropping the frames enough to assemble the finished(stabilized and lower resolution) frame from inside a full(unstabilized) frame.

FIG. 3(a)-FIG. 3(g) illustrate frames 200(a)-200(g) shown in FIG.2(a)-FIG. 2(g) rendered on a player 300 in conventional fashion. Eachportion of actual scene 100 captured in any particular frame 200 _(x) isrendered in normalized fashion to the orientation of player 300 whichimparts apparent jitter in response to changes in the field-of-view andother imaging parameters when the content of the respective frame 200_(x) is recorded. The viewer is then subjected to the jitter as theyview the rendered aligned frames 200 _(x) in player 300.

FIG. 4 illustrates a parsing of actual scene 100 illustrating theorientation and positioning of frames 200 _(x) illustrated in FIG.2(a)-FIG. 2(g) and FIG. 3(a)-FIG. 3(g). The orientation and positioningof frames 200 _(x) is simplified in that only rotational anddisplacement transformations are included. In other implementations, thetransformations may include scaling and skewing/distortions that wouldmake illustration more complicated to demonstrate and visualize (andrequire additional processing to composite as further described below)without further adding to an understanding of the present invention.

FIG. 5(a)-FIG. 5(g) illustrate the series of frames shown in FIG.2(a)-FIG. 2(g) rendered on an image presentation device 500. Imagepresentation device 500 provides an enhanced display area that is largerthan the physical dimensions of each frame 200 _(x). FIG. 5(a)-FIG. 5(g)render each frame 200 _(x) on image presentation device 500 within avirtual space that encompasses the expanse of the entire series offrames, and each frame 200 _(x) is rendered in relative context toframes 200 _(x) that have been previously rendered, and as is furtherexplained herein, using a meta-context map to position each frame 200_(x) on image presentation device 500. FIG. 5(a)-FIG. 5(g) illustrateuse of a 100% dwell (e.g., frame 200 persistence) wherein each frame 200_(x) remains rendered on image presentation device 500 as additionalsuccessive frames 200 _(x) are added.

FIG. 5(a) illustrates a rendering of frame 200(a) on image presentationdevice 500. FIG. 5(b) illustrates a rendering of new frame 200(b) andpreviously rendered frame 200(a) on image presentation device 500. FIG.5(c) illustrates a rendering of new frame 200(c) and previously renderedframes 200(a)-200(b) on image presentation device 500. FIG. 5(d)illustrates a rendering of new frame 200(d) and previously renderedframes 200(a)-200(c) on image presentation device 500. FIG. 5(e)illustrates a rendering of new frame 200(e) and previously renderedframes 200(a)-200(d) on image presentation device 500. FIG. 5(f)illustrates a rendering of new frame 200(f) and previously renderedframes 200(a)-200(e) on image presentation device 500. FIG. 5(g)illustrates a rendering of new frame 200(g) and previously renderedframes 200(a)-200(f) on image presentation device 500, in which actualscene 100 is reproduced (because of the 100% dwell). In someembodiments, it is desirable to provide some indicator, for example areticule or border, of a perimeter of the most recently rendered frame200 _(x), or provide some other mechanism (e.g., a change in intensityor other display property of the most recently rendered frame such asmay suggest a flash of operating a light-source often associated withimage capture) to highlight the appearance of new content. In this waysuch an indicator provides a context of the currently referenced imageand/or aids the view by providing an eye-catching “follow along”indication as a guide to make it easier for the viewer, particularlyadvantageous when synchronizing with audio but useful in other contextsas well. In other implementations, it is desirable to build a borderthat includes an outline perimeter of all currently rendered frames 200_(x) within image presentation device 500. In some implementations, bothan indicator and a border are used. Rendering of frame 200(a)-frame200(g) is relatively easy in this case since content alignmentattributes (e.g., the common anchors and shared references and the like)are visible in consecutive frames.

In the preferred embodiment, apparent jitter is “absorbed” by locatingeach frame 200 _(x) at the proper relative location to other frames 200_(x), with the indicator providing a virtual jitter as it rotates andtranslates and otherwise shifts during rendering of successive frames.Preferred embodiments of the present invention allow the system tomanage this apparent jitter in new-found ways, for example the systemmay enable some jitter to be depicted by actually distorting thelocation and orientation of anchors of individual frames 200 _(x) duringrendering to deviate from their parsed locations and orientations. Thisenables the system to manage how much of the dynamic meta-motion of theimager (previously classified as apparent jitter) to pass on to theviewer as for some series the meta-motion imparts energy as it moreaccurately conveys information relating to the imaging process. (Forexample, a videographer that is running to escape some peril.) Allowingimage presentation device 500 to impart a desired quantity of the jitteronto frame 200 _(x) when rendering location/orientations allows theproducer and/or the viewer to manage the jitter without it being toodistracting and interfering with absorption of the overall message.Induced jitter is recreated or enabled by displacing content alignmentattributes from actual locations during rendering.

FIG. 6 illustrates a match between the parsing of actual scene 100 andrendering of frames 200 _(x) in their proper location and orientation toreproduce actual scene 100 from the individual frames of the series offrames illustrated in FIG. 2(a)-FIG. 2(g). In some embodiments of thepresent invention, an aggregation of the locations, sizes, andorientations of frames 200 _(x) serve as a context map.

FIG. 7(a)-FIG. 7(k) illustrate the series of frames shown in FIG.2(a)-FIG. 2(g) rendered on an image presentation device in which eachindividual frame has a limited dwell time. In these illustrations, whena frame has been rendered for the indicated period, it is dropped. Thusthe frame appears suddenly, persists for the dwell time, and thendisappears suddenly as it dropped. In some embodiments, it may be thatthe appearance and/or the disappearance occur gradually, as in a fade-inor fade-out of the frame or frames (as the effect may extend overseveral frames), and the fade-in and fade-out times need not be thesame.

FIG. 7(a)-FIG. 7(k) illustrate the series of frames shown in FIG.2(a)-FIG. 2(g) rendered on an image presentation device 500. Imagepresentation device 500 provides an enhanced display area that is largerthan the physical dimensions of each frame 200 x. FIG. 7(a)-FIG. 7(k)render each frame 200 _(x) on image presentation device 500 within avirtual space that encompasses the expanse of the entire series offrames, and each frame 200 _(x) is rendered in relative context (e.g.,responsive to one or more shared anchors) to frames 200 _(x) that havebeen previously rendered, and as is further explained herein, using ameta-context map to position each frame 200 _(x) on image presentationdevice 500. FIG. 7(a)-FIG. 7(k) illustrate use of a 3-frame dwell (e.g.,a particular frame 200 _(i) persists for the time taken to render 3frames) wherein each frame 200 _(x) remains rendered on imagepresentation device 500 for a limited time as additional frames 200 _(x)are added.

FIG. 7(a) illustrates a blank image presentation device 500 prior to anyrendering of any frame 200 _(x). FIG. 7(b) illustrates a rendering offrame 200(a) on image presentation device 500. FIG. 7(c) illustrates arendering of new frame 200(b) and previously rendered frame 200(a) onimage presentation device 500. FIG. 7(d) illustrates a rendering of newframe 200(c) and previously rendered frames 200(a)-200(b) on imagepresentation device 500. FIG. 7(e) illustrates a rendering of new frame200(d) and previously rendered frames 200(b)-200(c) on imagepresentation device 500 (frame 200(a) is no longer rendered). FIG. 7(f)illustrates a rendering of new frame 200(e) and previously renderedframes 200(c)-200(d) on image presentation device 500 (frames200(a)-200(b) are no longer rendered). FIG. 7(g) illustrates a renderingof new frame 200(f) and previously rendered frames 200(d)-200(e) onimage presentation device 500 (frames 200(a)-200(c) are no longerrendered). FIG. 7(h) illustrates a rendering of new frame 200(g) andpreviously rendered frames 200(e)-200(f) on image presentation device500 (frames 200(a)-200(d) are no longer rendered). FIG. 7(i) illustratesa rendering of previously rendered frames 200(f)-200(g) on imagepresentation device 500 (there are no new frames 200 x to be renderedand frames 200(a)-200(e) are no longer rendered). FIG. 7(j) illustratesa rendering of previously rendered frame 200(g) on image presentationdevice 500 (there are no new frames 200 _(x) to be rendered and frames200(a)-200(f) are no longer rendered). FIG. 7(a) illustrates a blankimage presentation device 500 after all renderings of frames 200 _(x)have been performed and dropped based upon the desired dwell.

FIG. 8(a) and FIG. 8(b) illustrate rendering of individual frames 200_(x) from the series of frames shown in FIG. 2(a)-FIG. 2(d), with a 3frame dwell, and including an outline border 800 of currently renderedframes. As noted herein, some embodiments may include outline border 800identifying the aggregation of all, or a portion, of currently renderedframes 200 _(x). FIG. 8(a), similar to FIG. 7(d) includes a rendering offrames 200(a)-200(c), on image presentation device 500, with theinclusion of outline border 800 tracing out the perimeter of frames200(a)-200(c). FIG. 8(b), similar to FIG. 7(e) includes a rendering offrames 200(b)-200(d), on image presentation device 500, with theinclusion of outline border 800 tracing out the perimeter of frames200(b)-200(d).

FIG. 9(a) and FIG. 9(b) illustrate rendering of individual frames 200 xfrom the series of frames shown in FIG. 2(a)-FIG. 2(d), with a 3 framedwell, and including a current frame indicator 900 for the most recentlyrendered frame. As noted herein, some embodiments may include currentframe indicator 900 identifying the most recently rendered frame 200 x.FIG. 9(a), similar to FIG. 7(d) includes a rendering of frames200(a)-200(c), on image presentation device 500, with the inclusion ofcurrent frame indicator 900 tracing out the perimeter of frame 200(c).FIG. 9(b), similar to FIG. 7(e) includes a rendering of frames200(b)-200(d), on image presentation device 500, with the inclusion ofcurrent frame indicator 900 tracing out the perimeter of frame 200(d).

Outline border 800 and current frame indicator 900 are eachindependently selectable enabling a viewer to customize the presentationof rendered frames using image presentation device 500. Outline border800 is shown in solid while current frame indicator is shown using 50%opacity. Other arrangements and configurations of these structures arewithin the scope of the present invention.

FIG. 10 illustrates a schematic block diagram of a display system 1000.Display system 1000 receives an electronic file 1005 containing datarepresenting the series of frames. Electronic file 1005 may includediscrete frames 200 _(x) or some other format that enablesproduction/capture of discrete frames 200 _(x). A parser 1010 receiveselectronic file 1005 and produces/captures a set 1015 of discrete frames200 _(x). A compositor 1020 analyzes set 1015 of discrete frames 200_(x) and establishes a contextual temporal/geographic map 1025 thatplaces each discrete frame 200 _(i) within a virtual space andidentifies absolute/relative offsets, rotations, and other axial andspatial transformations of one discrete frame 200 _(m) to anotherdiscrete frame 200 _(n), as well as providing any scripting and otheravailable customizations. A rendering engine 1030 uses contextualtemporal/geographic map 1025 to render set 1015 of discrete frames 200_(x) on image presentation device 500 and to generate a replica ofactual scene 100. A controller 1035 is coupled to parser 1010,compositor 1020, and rendering engine 1030 to direct, coordinate, andoversee the conversion of electronic file 1005 to replicating actualscene 100. Controller 1035 is preferably a processor implementingmachine instructions retrieved from a memory that direct the actions andprocesses described herein.

One or more of parser 1010, compositor 1020, and rendering engine 1030may be partially or wholly automated, depending upon the implementationspecifics of the data files and desired output. Similarly, one or moreof parser 1010 and compositor 1020 may be partially or wholly manual.

Compositor 1020 in some embodiments is an editing/production system thatenables an editor/producer to review set 1015 of discrete frames 200_(x) and to identify overlapping anchor\reference points in multipleframes to use in melding discrete frames 200 x together. Not alldiscrete images 200 _(x) may share a common anchor/reference point, andit is not always the case that an anchor/reference is visible, orincluded within the frame itself. The anchor/reference points enablecompositor to transform individual ones of discrete frames 200 _(x) toalign frames consistent with common anchors/references. For example, inframe 200(a), frame 200(b), and frame 200(c), the vehicle may serve asan anchor for these three frames 200 _(x). Additionally, for frame200(a) and frame 200(b), the building may also be used.

In some cases, automatic feature recognition and matching along withmultiaxial/multidimensional scaling and transformation may simplifyand/or automate the compositing process. This is particularly true when,as in the preferred embodiments, set 1015 of discrete frames 200 _(x)from electronic file 1005 are related and share one or moretemporal/geographic characteristics. Display system 1000 enhances thoseshared characteristics when rendering discrete frames 200 _(x) usingcontextual temporal/geographic map 1025.

FIG. 11 illustrates a flowchart for a display process 1100. Displayprocess 1100 includes a parsing step 1105 to operate on electronic file1005 and produce set 1015 of discrete frames 200 _(x). Parsing step 1105may be simple or complicated depending upon the nature of electronicfile 1005 and what is required to identify and produce set 1015.

Following parsing step 1105 is a compositing step 1110 to operate on set1015. During compositing step 1110, common anchors/reference points areidentified in discrete frames 200 _(x) and used to create contextualtemporal/geographic map 1025. Contextual temporal/geographic map 1025identifies the absolute and relative transformations (which includesdisplacements) of discrete frames 200 _(x) to make a replica of actualscene 100.

Following composing step 1110 is a rendering step 1115 that renders set1015 on image presentation device 500 using contextualtemporal/geographic map 1025. Rendering step 1115 places individualdiscrete frames 200 _(x) at their appropriate absolute and relativelocations in the virtual space and with respect to each other.

In operation, discrete frames are displayed sequentially in the displayfield, such as within the viewable area of image presentation device500. Each frame 200 _(x) is positioned in the display field in aposition, size, and orientation corresponding to the position andcondition of the camera at actual scene 100 at the time that theparticular frame 200 _(x) was recorded. Each frame 200 _(x) may beallowed to persist for a time in the display field, to produce a“collage” that is an aggregation of the overlapping frames,progressively building up a replica of the actual scene to provide acontinuously updated and relatively stable representation of therendered scene in the display field. Any background region in thedisplay field that is not covered by a frame is filled with a neutralbackground color, or pattern or the like, signifying a region of “nophotographic information” in the display field. The portion of the scenecaptured by each individual frame is placed in the correct positionrelative to the larger scene in the display field (and comprises thelatest update to the larger scene). The position of each frame in thedisplay field corresponds to the position of the camera relative to theactual scene at the instant in time that the frame was captured. Thesequential location where each frame appears in the display field maytherefore jump around, sometimes significantly, according to the motionsof the camera field-of-view. The eye of the viewer of the playback canreadily track, and rapidly follow these motions relative to therelatively stable background scene in the display field. The eye does,in this way, easily, rapidly, and naturally follow, capture and recordeach new piece of visual information, together with its spatial locationrelative to the rest of the scene, much as it would (seeminglyeffortlessly) in real life at the real scene. Note in addition, that themotions of the camera may be fully represented (rather than beingsuppressed) in the display, and add an additional dimension to theresult, helping to convey an even more comprehensive visual impressionof the scene, including the motions of the camera, and the imputedexperience of the photographer.

The improved display method embodiments described herein includeimprovements over existing methods by more naturally engaging and morefully utilizing the natural visual capacity and seemingly effortlessperformance of the eye of the viewer to capture, comprehend, compensatefor, and “see through” even wildly unsteady motions of the camera, thesubjects, the viewer, and the eye. Note that in some implementations,rendering frames with desired colocations of identified referenceattributes may result in an aggregate composite that has “gaps,”“holes,” or “seams” of missing content. In some cases, a look-aheadfeature may fill in such missing content, at the user discretion, and inother cases, there will be missing content in the aggregate composite.In other cases, such as is typical for the preferred embodiments wherethe serial/sequential frames are temporal representations of the actualenvironment, there are elements in the frames that will not becompletely consistent. For example, two background people walking indifferent directions. Depending upon what the editor has chosen to useas a reference, different content elements may appear fragmented,indistinct, fuzzy, or become blended into the background, or disappear.Some embodiments will replace and/or remove inconsistentbackground/elements. A user, choosing to set a different reference, mayproduce a markedly different aggregate composite.

In many of the descriptions provided herein, the discussion explicitlyaddresses the problem of stabilizing a series of frames of a(stationary) scene and ignores the processes for display of normal,intentional, or smooth motions and changes such as panning, zooming inor out, traveling, following a moving object, changing scene, and thelike, that would occur in any motion picture. Embodiments of the presentinvention of course may be adapted to address such scenarios.

Embodiments of the present invention embody a new display paradigm. Thenew display paradigm represents a departure from a system that seeks toreject the effect of camera motion in the photographic display, to asystem that embraces the camera motion as an integral part of thephotographic display. Stated in contrasting intuitive terms, whereas theconventional paradigm keeps the frame still and lets the image jumparound (image jitter is present), the new paradigm seeks to keep theimage still and lets the frame jump around (image jitter is absent). Thenew display paradigm is able to integrate the camera motion as a part ofthe photographic picture, or the real-world visual experience that thedisplay is trying to convey to the viewer, where the camera motion isable to be fully represented independently as part of the picture,without apparent image jitter. The preferred embodiments create arealistic impression of the photographed subject as intended by thephotographer; and portrays a clear picture of what the photographerexperiences, including the user-selected, variably compensated cameramotion. The new display paradigm is able to embrace the camera motion asan integral part of the photographic picture, or the real-world visualexperience that the display is trying to convey to the viewer, where thecamera motion is able to be fully represented independently as part ofthe picture. Further, in some preferred implementations each customizedelement to be rendered is a result of a non-destructive process thatpreserves the original content. Non-destructive embodiments may eitherapply a modifying display parameter to the original content, or employaltered duplicate content, or use other content preservingmethodologies.

In a preferred embodiment, full content of all frames are preserved andnever modified. The original data is always available for closeexamination without loss or any compromise of the original photographicinformation (only display attributes can be modified). This would makethe system especially useful in presenting customized information, suchas forensic or other evidentiary or legal documentary evidence, as incourt, with assurance that the photographic information is the true andwhole original. The original data can then be viewed/analyzed in anyfashion with options available in a suitable display/editor system.

In preferred embodiments, the storage format is able to handle multipleformats. Since the user is free to format and store one or more framesany way he wants (not by modifying the content, but by modifying andsaving the display parameters) he can create and save multiple formatsfor display in different contexts. He also can always go back anddisplay them in a new format (e.g., still or motion protocol) any timein the future.

The systems and methods are preferably implemented using amicroprocessor executing program instructions from a memory, theinstructions causing the apparatus to perform as described herein.

In the description above, the rendered replica of actual scene 100includes all discrete frames 200 _(x). In some implementations, a scenemay include or exclude any or all frames in data set, including framesin the sequence which may have not have been “displayed” yet, forexample, some implementations enable a use of a “look-ahead” and analyzeupcoming frames to build up background or to establish shared anchorsand reference points or other elements to be rendered “out of turn” toaid in understanding or to fill “blank” areas.

Some embodiments of the present invention use the imager as apreprocessor when the picture is taken, or in a postprocessor to preparefor display. Motion data input source can be in any form: actualmeasurement in the camera via accelerometers, gyroscopes in the camera,calculated through comparisons of image regions within the displayedframes themselves, or data transmitted in a data file or data stream,incorporated in or accompanying the video signal; digital techniquesrelying on the content of the images themselves to infer calculatedcamera motion, use of an editor program or software to permit humaninterpreting and manipulation, and/or combinations thereof.

The editor/player described herein may be capable of setting andadjusting all display parameters. The display is preferably optimizedfor maximum effect based on authorship, special knowledge of the scene,judgment, or artistic sense, or to adjust positioning, cropping,panning, zooming, playback rate and timing, intentional motion, colorcorrection, editing individual frames, and the like, to help focus onthe intended subject of interest, introduce dressings such as currentframe highlighting, frame outlining, selection of frames to display, orselect frames to stay on top of other frames, or frames to discard tobest portray the retained base scene as desired. Display forward orbackward, stepwise, or individually, or on a slider back and forth atwill. Display of all parameters, adding titles, text and identificationnumbers and marks as well as other features and elements describedherein. The editor is able to create new modified video output or outputfile, which retains all display information.

Display factors may be included in a header or other data block or othermeta-information associated with a new standardized universal dataformat. Display factors can be X- and Y-displacements or speeds oraccelerations, Z-zoom displacements, time of frame or time delay sincelast frame, Intentional motion (panning, zooming, and the like) or acombination of these including titles, text, notes, dates, help, and thelike.

Output may include any form of video to be displayed, such as forexample: direct display on computer screen or other computer displaydevice; direct video signal to a video display device, and/or conversionto a conventional or proprietary video data file for storage andsubsequent display or use.

In the preceding discussion, to simplify understanding of certainaspects of the present invention, certain parameters andcharacterizations of embodiments of the present invention includeconstants that, in a more general description of the present invention,could include variable values. For example, it has already beendescribed that a rendered frame may have a different size or othertransformation as compared to an input frame from a data source.

As a matter of convenience, it was described that the rendered frame iscomposited in a display environment that is larger than frame sizes. Itis understood that this description includes a condition that renderedcontent of the entirety of the display environment is visible. This istrue if a viewport matches the display environment. The viewport is thatportion of the display environment (e.g., the virtual aggregatecomposite of renderable frames) that identifies content to be actuallyrendered (which may not encompass an entire expanse of the virtualaggregate composite). In some embodiments, the viewport is a staticdescription in the contextual map, and in other embodiments, theviewport is dynamic. A dynamic viewport may transform (e.g., scale,change, translate (e.g., change position and the like), rotate, andotherwise modify/define an area of renderable content. In embodiments ofthe present invention, ultimately rendered content within the viewportis presented on a display device, and the image data for the displaydevice are often visualized in response to being written or otherwisetransferred into a display buffer or the like. The viewport may bemapped to such a display buffer in order to visualize the renderedcontent within the viewport. In such cases, it is possible that somerendered frames do not completely fit within the viewport and may bewholly or partially cropped, or in some cases, completely outside theviewport and not rendered.

The characteristics of the viewport may be created/edited similarly tothe creation of the contextual map defining the aggregate composite,such as by an editing process. This editing process may be manual, suchas could be the case when an editor defines the viewport prior torendering any part of the aggregate composite. In other cases, theediting process is automatic based upon parameters designed to achievethe desired effect for the viewport, and is defined prior to anyrendering. In other cases the viewport may be defined/modified duringthe playback/rendering process. A user, for example operating acontroller similar to a game controller or the like, interacts with thesystem to manipulate the viewport (e.g., scale, location, rotation, andthe like). Other times, the post-rendering editing process for theviewport may be responsive to user/device settings. For example, theuser may have defined particular patterns (e.g., faces, structures,machines, and the like) and the editing process defines the viewport toinclude/exclude matching/mismatching patterns automatically. Themanual/automatic editing process may be configured to override apredetermined viewport definition when necessary or desired.

The system and methods herein have been described in general terms as anaid to understanding details of preferred embodiments of the presentinvention. In the description herein, numerous specific details areprovided, such as examples of components and/or methods, to provide athorough understanding of embodiments of the present invention. Oneskilled in the relevant art will recognize, however, that an embodimentof the invention can be practiced without one or more of the specificdetails, or with other apparatus, systems, assemblies, methods,components, materials, parts, and/or the like. In other instances,well-known structures, materials, or operations are not specificallyshown or described in detail to avoid obscuring aspects of embodimentsof the present invention.

Reference throughout this specification to “one embodiment”, “anembodiment”, or “a specific embodiment” means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention and notnecessarily in all embodiments. Thus, respective appearances of thephrases “in one embodiment”, “in an embodiment”, or “in a specificembodiment” in various places throughout this specification are notnecessarily referring to the same embodiment. Furthermore, theparticular features, structures, or characteristics of any specificembodiment of the present invention may be combined in any suitablemanner with one or more other embodiments. It is to be understood thatother variations and modifications of the embodiments of the presentinvention described and illustrated herein are possible in light of theteachings herein and are to be considered as part of the spirit andscope of the present invention.

It will also be appreciated that one or more of the elements depicted inthe drawings/figures can also be implemented in a more separated orintegrated manner, or even removed or rendered as inoperable in certaincases, as is useful in accordance with a particular application.

Additionally, any signal arrows in the drawings/Figures should beconsidered only as exemplary, and not limiting, unless otherwisespecifically noted. Furthermore, the term “or” as used herein isgenerally intended to mean “and/or” unless otherwise indicated.Combinations and/or separation of components or steps will also beconsidered as within the scope of the present invention, includingimplementations where the terminology is foreseen as rendering anability to separate or combine is unclear, provided undueexperimentation is not required.

As used in the description herein and throughout the claims that follow,“a”, “an”, and “the” includes plural references unless the contextclearly dictates otherwise. Also, as used in the description herein andthroughout the claims that follow, the meaning of “in” includes “in” and“on” unless the context clearly dictates otherwise.

The foregoing description of illustrated embodiments of the presentinvention, including what is described in the Abstract, is not intendedto be exhaustive or to limit the invention to the precise formsdisclosed herein. While specific embodiments of, and examples for, theinvention are described herein for illustrative purposes only, variousequivalent modifications are possible within the spirit and scope of thepresent invention, as those skilled in the relevant art will recognizeand appreciate. As indicated, these modifications may be made to thepresent invention in light of the foregoing description of illustratedembodiments of the present invention and are to be included within thespirit and scope of the present invention.

Thus, while the present invention has been described herein withreference to particular embodiments thereof, a latitude of modification,various changes and substitutions are intended in the foregoingdisclosures, and it will be appreciated that in some instances somefeatures of embodiments of the invention will be employed without acorresponding use of other features without departing from the scope andspirit of the invention as set forth. Therefore, many modifications maybe made to adapt a particular situation or material to the essentialscope and spirit of the present invention. It is intended that theinvention not be limited to the particular terms used in followingclaims and/or to the particular embodiment disclosed as the best modecontemplated for carrying out this invention, but that the inventionwill include any and all embodiments and equivalents falling within thescope of the appended claims. Thus, the scope of the invention is to bedetermined solely by the appended claims.

What is claimed as new and desired to be protected by Letters Patent ofthe United States is:
 1. A computer implemented video playback methodfor displaying a series of input frames, wherein each input frame of theseries of input frames has an associated camera orientation information,a visible image content, and a frame boundary; wherein the cameraorientation is independently fully displayed, the visible image contentis displayed stabilized free from the disturbing effects of the cameraorientation motion, and no image content is lost by constructing anoutput video comprising a series of output display frames using adisplay frame larger than any input frame of the series of input framesand wherein each input frame of the series of input frames is placed ina position within an output display, wherein the camera orientation isdisplayed by the location and rotation of the input frame boundary asplaced within the output display frame; the visible image content isdisplayed in the output display frame free of the destabilizing effectsof the camera motion, and the camera motion is simultaneously depictedby the apparent motion of the input frame boundary of within the displayframe in the output series of display frames rendered as a moving visualimage stream; the method comprising the steps of: (a) placing a firstinput frame of the series of input frames within a first output displayframe of the series of output display frame, which positions the firstinput frame at a particular location relative to the first outputdisplay frame, and thereafter (b) determining the placement position todepict the position of the second input frame of the series of inputframes to be placed relative to the position of the first input frameresponsive to the camera orientation information associated with thesecond input frame relative to the first input frame, (c) placing thesecond input frame in the second output display frame of the series ofoutput display frame in the position determined in step (b), (d)rendering the series of output display frames as a moving visual imagestream wherein, since an instantaneous camera orientation is depicted asthe apparent location and rotation of a particular frame boundary withinits associated output display frame, the apparent camera motion isdepicted as the apparent motion of the frame boundary within the seriesof output display frames rendered as a moving visual image stream;wherein said rendering step preserves a portion of any previouslyrendered content including the first frame; wherein said portion of anypreviously rendered portion is outside the boundary of said second frameand wherein said portion of any previously rendered content has a dwellperiod, with said portion of any previously rendered content removedfrom the display frame after having been rendered in the display framefor a time responsive to said associated dwell period.
 2. The videomethod of claim 1 where the input frame associated camera orientationinformation is provided implicitly in the visible content of each frameand are inferred by analysis of the image content and the apparentpositions of selected visible features relative to the borders of eachfirst input frame and second input frame, features that are presumed tobe stationary in an intended frame of reference.
 3. The video method ofclaim 2 wherein the selected feature defines the intended frame ofreference as the feature itself.
 4. The video method of claim 1 whereinthe dwell period is specified in a number of frames.
 5. The video methodof claim 4 wherein the dwell period includes three frames.